(n.d.)
- Goal / research question
- Test whether an untrained LLM can identify and label causal claims in qualitative interview “stories” well enough to be useful, compared with human expert coding (a criterion study).
-
Focus is on validity/usefulness of causal-claim extraction, not causal inference.
-
Core framing: causal mapping vs systems modelling
- In systems mapping, an edge \(X \rightarrow Y\) is often read as “\(X\) causally influences \(Y\)”.
- In causal mapping (as used here), an edge means there is evidence that \(X\) influences \(Y\) / a stakeholder claims \(X\) influences \(Y\).
-
Output is therefore a repository of evidence with provenance, not a predictive system model.
-
“Naive” (minimalist) causal coding definition
- Deliberately avoids philosophical detail; codes undifferentiated causal influence only.
- Does not encode effect size/strength; does not do causal inference; does not encode polarity as a separate field (left implicit in labels like “employment” vs “unemployment”).
-
Coding decision reduced to: where is a causal claim, and what influences what?
-
Data and criterion reference
- Corpus from a QuIP evaluation (2019) of an “Agriculture and Nutrition Programme”.
- Dataset previously hand-coded by expert analysts (used as a criterion study).
-
Validation subset: 3 sources, 163 statements, ~15 A4 pages.
-
Extraction procedure (AI as low-level assistant)
- Implemented via the Causal Map web app using GPT‑4.0.
- Temperature set to 0 for reproducibility.
- AI instructed to produce an exhaustive, transparent list of claims with verbatim quotes; synthesis is done later by causal mapping algorithms.
- Exclusions: ignore hypotheticals/wishes.
-
Output per claim: statement ID + quote + influence factor + consequence factor.
-
Two validation variants
- Variant 1 — open coding (“radical zero-shot”)
- No codebook; includes an “orientation” so the AI understands the research context.
- Uses a multi-pass prompting process (initial extraction + revision passes).
-
Variant 2 — codebook-assisted (“closed-ish”)
- Adds a partial codebook (most-used top-level labels from the human coding).
- Uses hierarchical labels
general concept; specific concept.
-
Validation metrics and headline results
- Precision (human-rated, four criteria): correct endpoints; correct causal claim; not hypothetical; correct direction.
- Variant 1: 180 links; perfect composite score (8/8) for 84% of links.
- Variant 2: 172 links; perfect composite score (8/8) for 87% of links.
-
Recall (proxy): compared link counts vs the human-coded set (acknowledging no true ground truth because granularity is underdetermined).
-
Utility check (overview-map similarity)
- Detailed maps differ (expected in qualitative coding).
-
When zoomed out to top-level labels and filtered to the most frequent nodes/links, AI and human overview maps are broadly similar.
-
Scope limits / risks
- Small sample; single (relatively “easy”) dataset; informal rating process.
- Label choice/consistency remains a major source of variation; batching can introduce inconsistency across prompts.
- Suitable for mapping “how people think” and building auditable evidence sets; not suitable for high-stakes adjudication of specific links without checking.